Anchor 2005

Flow tracking is an integral aspect of many network-related applications, including intrusion detection, network administration, per-flow billing, and network engineering. While some applications do not require precise tracking of every packet or flow and successfully employ statistical sampling or approximations to achieve an acceptable level of accuracy, many others, especially in the security domain, do not have such flexibility. Faced with rapidly growing number of networked devices, increasing Internet usage, and rising network bandwidth, precise flow tracking has become a very difficult problem with system memory as the main bottleneck. In this paper, we look at memory performance when tracking a large number of flows, show that finding an existing flow in memory often requires multiple key comparisons, and examine ways to reduce the number of comparisons for existing flows and avoid them altogether for the majority of new flows. Additionally, we propose a novel scheme called Predictive Placement that reduces the number of searches requiring three or more comparisons from 4.87% with a linked list implementation to just 0.0027% of all lookups. These results are based on our simulations with widely used long packet traces from CAIDA. Our approach relies on a Bloom filter derivative that encodes each element’s location using multiple bitmaps. The accuracy of the scheme depends on the number of bitmaps employed and on their load factor.